Neural Variational Inference and Learning in Belief Networks

•We introduce a simple, efficient, and general method for training directed latent variable models. – Can handle both discrete and continuous latent variables. – Easy to apply – requires no model-specific derivations. •Key idea: Train an auxiliary neural network to perform inference in the model of interest by optimizing the variational bound. – Was considered before for Helmholtz machines and rejected as infeasible due to high variance of inference net gradient estimates. •We make the approach practical using simple and general variance reduction techniques. •Promising document modelling results using sigmoid belief networks.